Next Linux Frontier: Everything Else

By Doc Searls

Linus Torvalds has been pointing in an embedded direction for a long time, starting with his own career path. In 1997, Linus moved from Finland to Silicon Valley, where he went to work for the famously secretive Transmeta. While Linus has been extremely discreet about Transmeta's work, he has been more forthcoming about his interest in appliances and embedded applications. Hardly a speech or an interview goes by without mention of embedded Linux possibilities.

In fact, Linus helped develop Transmeta's own Mobile Linux, which is ROM-based and customized for Transmeta's own TM3120 Crusoe processor. Here's what Linus said about it when I talked with him on Transmeta's launch day, earlier this year: "It's the standard 2.3 kernel with some power management stuff done on it, but anybody can see what that is. There is the compressed file system that I made part of the standard kernel not long ago. And that's really the question of ... you have a very limited amount of ROM. You want to fit Netscape in there, too. And you want to page things in from the ROM. You obviously want to compress it. That's number one. But how do you compress it so you can still do random seeks and paging in things? Those are the kinds of questions I spent a fair amount of time thinking about. What's the good way of doing this? What fits in and works well? These are not fundamental design shifts."

They aren't fundamental design shifts because Linux is already well-adapted — or adaptable — to the embedded application world.

Lately the hot embedded topic is "appliances." Mobility is another hot issue. Both suggest that the embedded world is about physically small things. But to embedded systems, size isn't the issue. Purpose is. A massive telephone switching system may have a specialty no larger in operational scope than a pager. Both are special. Neither is general.

For all of computing's history, the difference between "special" and "general" operating systems was not just a matter of scope. It was a massive difference in kind. Nearly every attempt to leverage one to the other failed. Apple, AT&T and Microsoft, all successful at creating large general purpose operating systems, failed (or were at least frustrated) in their attempts to make special ones. Novell also failed to recast NetWare — an extremely successful special purpose network operating system — as a general purpose competitor for Windows NT.

So for decades the embedded world has largely been the province of well-niched embedded OS companies such as Wind River and Mentor Graphics. The market was hardly small. Today Wind River <www.windriver.com> alone has sales pushing $200 million/year, and a market cap of nearly $3 billion. But for the most part it was safe from Microsoft and everybody else who made a general purpose operating system.
When you have thousands of developers working on millions of lines of code, it is absolutely critical to support both clean interfaces and the ability to rip out somebody else's code and still have your code work. A happy consequence of the open source model is that Linux evolved, by necessity, to be more modular, and to support cleaner interfaces, than a monolithic design that assumes less anarchistic control. As a result, Linux kernels as small as 300KB can not only boot up with networking support, but serve web pages as well. This is really exciting to people who want to put web servers (and administrative consoles) everywhere.

You can also use the same APIs for development and deployment. While this is not a new goal, the reality of a system that (1) works, (2) people actually like to program, and (3) supports tons of content, has expanded the realm of embedded system development from a highly specialized art to something any Linux programmer can manage. This dramatic expansion of the developer base is going to have a profound impact on the embedded system market.

It is always difficult to say who has the "best" version of embedded linux. Our take is that even in embedded, There Is Only One Linux. Namely, we're building configuration tools (open source, of course), which make it possible to take a stock Red Hat 6.2 distribution, configure out 99.99% of stuff that's not relevant for an embedded system, and voila, it's still Red Hat. Others are taking different approaches, such as offering many different versions of Linux for many different markets, but we believe in the One Source approach. It's more maintainable, and we believe it will better server a larger number of developers over the long run.

— Michael Tiemann, Chief Technology Officer, RedHat

But now Linux is invading this world in a huge way. Suddenly old real-timers are either adopting Linux or being adopted by Linux companies. James Ready, the embedded pioneer who founded Ready Systems a few years back, now runs MontaVista <www.mvista.com>, which launched HardHat Linux last year. RedHat acquired Cygnus last Fall, then added WireSpeed in June of this year. (The sidebar is by Michael Tiemann, the Red Hat CTO who founded Cygnus.) Last summer Caldera, an early Linux distribution leader, spun off Lineo as an embedded Linux company. Both Lineo and MontaVista have substantial venture capital backing, and are themselves in aggressive acquisition modes. With all this movement in a short time, traditional embedded publications and events are suddenly abuzz with Linux coverage.

This buzz isn't wishful thinking about new dot-com possibilities. This is about building a distributed computing world because Linux makes it easy and there are a lot of Linux architects and builders looking for fun new stuff to make.

Here's John Bork of Intel, which is building its new Dot.Station Web appliances and set-top boxes on a combination of embedded Linux and customized Mozilla : "We just take the source off the Net, pull out the stuff we don't need, seeing what libraries are used and not used, and constraining ourselves from there. See, the biggest challenge we face is getting the OS, the browser, the plug-ins and a few applets into eight megabytes of flash. So a lot of our work has gone into analyzing what pieces we need, and how we skinny this thing down."

Kerbango's radio and Intel's Dot.Station are just two among the first wave of Net-native appliances built using embedded Linux (the Dot.Station also uses Mozilla).
This work has a lot of appeal. "I enjoy working on servers like I did at my last job," says one Linux programmer at Kerbango, a start-up using embedded Linux in its Web radios, "But I love working on cool new stuff." In the past, learning a whole new operating system and all its tools was expensive in both time and money. Not so with Linux. An embedded application might be extremely arcane, but the operating system knowledge required is not. That can hold down NRE (non-recoverable engineering) costs to a highly significant degree.

"Skinnying down" Linux, however, is not an easy job. That's why companies like Lineo and MontaVista are in business. Here's Kim Clark, VP Engineering at Lineo:

"The first thing you need to understand is, what are all the packages? Which ones do I need and which ones don't I need? What are the dependencies associated with each of them? Choosing one package may require choosing two or three other packages. But they also might conflict with packages from another module. So you have this hairy and complicated task of figuring out all the things that need to be there."

Among Lineo's tools is one called "LIPO," which goes in and, well, sucks out all the fat. "The shared libraries are big and fat," Clark says. "If you have several modules sharing libraries, each with a copy of that library, you've got a bigger footprint. We cut fat out of that. We also reduce the size of each library's' footprint. We do the same for Apache. By cutting out icon support, for example, you save 22k of memory. We make this possible by taking Apache, cutting it into microcomponents, giving developers the choice of using only the parts they need." (See sidebar.)
These screen shots shows Lineo's Embedix™ Target Wizard Selection and Configuration Interface at work, selectively enabling parts of both Linux and Apache.

Using tools like Lineo's helps Linux apply across a huge assortment of embedded devices, as this graphic points out.

This is Lineo's "Embedded Linux Market Spectrum," which differentiates between the space where its embedded product offerings (and acquisitions) apply, and the non-embedded PC/Server space, where they don't.

This is the Lineo view of the world, but it could be anybody's. None of the lines are definitive, which is why the boxes are shaded. Lyle Ball, VP Marketing for Lineo, says "The important thing is that, while the desktop and server areas are familiar territory, the rest of the world — the embedded device world — is not. There is a lot of existing expertise everywhere on this chart, but when you get down to the finished product level, that's where you've got the feudal system that Linux and these other developments begin to eliminate." David Rieves, Lineo's Director of Product Marketing, adds this: "People are going to have to think differently here — even people already in the embedded space. The glories of eight and sixteen bit systems are in the past, when that was the size needed for the required functionality of modern devices. They were cheap enough to build in without driving up cost too much. Now 32-bit hardware is cheap commodity stuff. So you hold overhead down in the OS, which is why Linux is so appealing. As hardware costs go down, interest in Linux goes up."